Image processing method and image processing apparatus

ABSTRACT

An image processing method includes inputting a first image to an image processing apparatus, the first image being associated with a first inspection result, inputting a second image to the image processing apparatus, the second image being associated with a second inspection result different from the first inspection result, inputting a third image to the image processing apparatus and setting a portion of the third image as an area to be subjected to image processing, by the image processing apparatus, based on the first image and the second image.

This application is a continuation application of U.S. patentapplication Ser. No. 16/657,099, filed on Oct. 18, 2019, which claimsthe benefit of Japanese Patent Application No. 2018-198743, filed onOct. 22, 2018, each of which is hereby incorporated by reference hereinin its entirety.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an image processing method and an imageprocessing apparatus for inspecting a target object by using imageprocessing.

Description of the Related Art

An image processing apparatus that performs inspection and measurementby using image processing is widely used for automation of inspection ina manufacturing line. There is a wide variety of inspection andmeasurement performed by this kind of image processing apparatus.

In this kind of image processing apparatus, an inspection/measurementtarget object is imaged by an image sensor, that is, a camera, imagedata of an inspection region is extracted from the obtained image, afeature value of the image within the inspection region isanalyzed/evaluated, and thus inspection/measurement that is aimed for isperformed.

For this kind of image processing apparatus, a preparation operationsuch as setting the inspection region needs to be performed beforeperforming the inspection/measurement processing. Typically, a user setsan inspection region corresponding to the purpose of theinspection/measurement by using a tool for setting the inspectionregion.

Meanwhile, there is a demand that the time required for the setting beshortened for improving the efficiency of the setting for small-lotproduction of a wide variety of products. In addition, due to increasein the complexity and precision of inspection/measurement, there is ademand that the inspection region be precisely set for only a portionthat should be the target of the inspection/measurement to preventerroneous detection.

The inspection target object has a different shape depending on theindividual, and there is also a case when the inspection target objectis imaged in a different scale, position, or orientation each time. Forexample, in some cases of imaging inspection of an application state ofa coating material on a product, the application state of a differentshape has to be inspected each time, and there is a possibility that theinspection region needs to be reset each time when inspection of highprecision is to be performed.

Conventionally, as a method of automatically determining the inspectionregion, an inspection region extraction method using binarization orcolor gamut extraction is known. In this method, a group of pixelscorresponding to a preset luminance range or a preset color gamut isextracted from an image, and the group of pixels is extracted orselected as the inspection region.

This conventional method is relatively effective in the case when thecontrast of luminance or color between a portion that is to be extractedas the inspection region, that is, the foreground, and the otherportion, that is, the background, is high. However, in the case when theforeground portion that is an inspection target is constituted byvarious luminance or various colors like the case of a printed board, orwhen a color close to that of a foreground portion is present in thebackground, it is difficult to precisely extract only the foregroundportion by binarization or color gamut extraction.

In view of the problem above, a method of obtaining an optimuminspection region by using a plurality of images and performing aprocess of enlarging, reducing, expanding, or contracting the inspectionregion is considered in Japanese Patent Laid-Open No. 2006-194607. Inthis method, an initial inspection region is set by using a plurality ofimages, then inspection/measurement processing is performed, and theinitial inspection region is enlarged, reduced, expanded, or contractedin various directions until the result of the inspection/measurementprocessing reaches a predetermined level.

In addition, a method of generating an outline of the inspection regionby using edge extraction to obtain an inspection region appropriate foran inspection target object in an inspection image is known, forexample, as disclosed in Japanese Patent Laid-Open No. 2013-191064. Inthis method, an edge of the inspection target object and a region forobtaining the edge are generated in advance from an image serving asstandard. Then, when the inspection image is input, an edge of theinspection target object is extracted from the inspection image by usingthe information, and the outline of the inspection region is generatedon the basis of the edge.

According to the inspection region generation method of Japanese PatentLaid-Open No. 2006-194607 described above, when determining theinspection region, all images have to be processed each time, whichrequires a large throughput and takes a long time for processing in asystem having a low processing capability. In addition, although aninspection region that fits the inspection target object better can beobtained by reducing the amount of enlargement/reduction orexpansion/contraction, the amount of calculation increases as the amountof enlargement/reduction or expansion/contraction is reduced, whichtakes a longer time for processing.

In addition, according to the inspection region generation method, theedge of the inspection target object can be extracted relatively nicelyby using an edge extracted by using the image serving as a standard, inthe case when a similar inspection target object is inspected each time.However, in inspection such as inspection of an application state of acoating material in which an application region of a different shape isimaged each time, there is a possibility that the extraction of edgefails. As described above, in inspection in which an image of adifferent shape is captured each time, there is a possibility that, forexample, a process of performing edge extraction and generating theoutline of the inspection region needs to be performed each time, whichincreases the image processing time.

SUMMARY OF THE INVENTION

According to a first aspect of the present invention, an imageprocessing method includes an identifiability value obtaining step ofobtaining, for each pixel address constituting an image plane, anidentifiability value for identifying to which of a first inspectionresult and a second inspection result the pixel address corresponds, byusing a feature value of each pixel of a first image obtained by imaginga first target object for which the first inspection result has beenobtained and the feature value of each pixel of a second image obtainedby imaging a second target object for which the second inspection resulthas been obtained, an inspection region generation step of generating aninspection region serving as a target of image processing by setting aportion of the image plane including the pixel address when the obtainedidentifiability value satisfies a specific condition as the inspectionregion, and an image inspection step of performing image processing forinspection on a partial image corresponding to the inspection regionamong a third image obtained by imaging a third target object.

According to a second aspect of the present invention, an imageprocessing apparatus includes an identifiability value obtaining portionconfigured to obtain, for each pixel address constituting an imageplane, an identifiability value for identifying to which of a firstinspection result and a second inspection result the pixel addresscorresponds, by using a feature value of each pixel of a first imageobtained by imaging a first target object for which the first inspectionresult has been obtained and the feature value of each pixel of a secondimage obtained by imaging a second target object for which the secondinspection result has been obtained, an inspection region generationportion configured to generate an inspection region serving as a targetof image processing by setting a portion of the image plane includingthe pixel address when the obtained identifiability value satisfies aspecific condition as the inspection region, and an image inspectionportion configured to perform image processing for inspection on apartial image corresponding to the inspection region among a third imageobtained by imaging a third target object.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a functional configuration of acontrol system of an inspection apparatus according to a first exemplaryembodiment of the present invention.

FIG. 2 is a flowchart illustrating a flow of inspection processingaccording to the first exemplary embodiment of the present invention.

FIGS. 3A to 3E are each an explanatory diagram illustrating a targetobject and how inspection is performed according to the first exemplaryembodiment of the present invention.

FIG. 4 is a flowchart of inspection region setting according to thefirst exemplary embodiment of the present invention.

FIGS. 5A to 5F are each an explanatory diagram illustrating a method ofdetermining an inspection region from image data according to the firstexemplary embodiment of the present invention.

FIGS. 6A to 6C are each an explanatory diagram illustrating an exampleof a setting screen for an inspection region according to the firstexemplary embodiment.

FIG. 7 is a flowchart of inspection region setting according to a secondexemplary embodiment of the present invention.

FIGS. 8A and 8B are each an explanatory diagram illustrating a method ofextracting an edge from image data.

FIG. 9 is a block diagram illustrating a configuration of a controlapparatus serving as a control system according to the first exemplaryembodiment.

DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments of the present invention will be described belowwith reference to attached drawings. To be noted, the configurationsshown below are merely examples, and, for examples, details thereof canbe appropriately modified by one skilled in the art within the gist ofthe present invention. In addition, numerical values described in theexemplary embodiments are values for reference, and should not limit thepresent invention.

The following exemplary embodiments relate to an image inspection systemthat performs, by using an image, external appearance inspection,inspection of whether presence/absence of an attached part, inspectionof application state, presence/absence, and application range of acoating material, or the like, and an inspection region setting methodof automatically or semi-automatically generating an inspection regionwhere image inspection is performed. This image inspection apparatus canbe suitably used for successively inspecting a large number of products,parts, members, or the like, serving as inspection targets automaticallyor semi-automatically regardless of the type or material thereof in afactory automation manufacturing line, or the like. Particularly, in thefollowing exemplary embodiments, an appropriate inspection region can beset by using a plurality of images captured in advance. Therefore, thiscan be particularly preferably applied to a case when, for example, theposition and shape of the target object in the image changes each timethe image is captured. Although there are a variety of purposes andinspection items for the external appearance inspection orpresence/absence inspection, the inspection region setting method of theexemplary embodiments can be preferably applied to any of theinspections.

First Exemplary Embodiment Processing Apparatus

FIG. 1 illustrates a functional configuration of an image processingapparatus according to a first exemplary embodiment. Hereafter, theimage processing apparatus may be also referred to as a processingapparatus for the sake of simplicity. A processing apparatus 100 of FIG.1 is constituted by hardware including an apparatus body 101, an imagesensor 102, a display apparatus 108, a storage device 109, and an inputdevice 110, and so forth.

The image sensor 102 is a device for capturing an image of a targetobject 103 which is, for example, a workpiece handled in a manufacturingline, and loading a color or monochromatic still or moving image intothe processing apparatus 100. For example, a digital camera havingsensitivity in approximately a visible light region can be used for theimage sensor 102. However, in the case of using a special imagedifferent from a visible light image such as an X-ray image, athermographic image, or a distance image, for the inspection, anappropriate sensor suitable for the mode of inspection may be used forthe image sensor 102. In the case when the processing apparatus 100 isprovided in a manufacturing line of a product, or the like, as an imageprocessing apparatus that performs image inspection, the image sensor102 is disposed at a specific imaging position in the manufacturingline.

The display apparatus 108 is a display device for displaying an imagecaptured by the image sensor 102, a result of inspection, and a graphicuser interface: GUI screen for inspection processing or settingprocessing. For example, a liquid crystal display or a display device ofa different type can be used as the display apparatus 108. The storagedevice 109 is a device for storing various setting information,inspection results, and so forth. The processing apparatus 100 refers tothe setting information in inspection processing, and examples of thesetting information include inspection region information and aninspection program, stored in the storage device 109. For example, ahard disk drive: HDD, a solid state drive: SSD, a flash memory, anetwork storage, or the like, can be used as the storage device 109. Theinput device 110 is a device that a user operates to input aninstruction in the apparatus body 101. For example, a mouse, a keyboard,a touch panel, a dedicated console, or the like, may be used as theinput device 110.

The hardware of the apparatus body 101 can be constituted by a computerapparatus including a central processing unit: CPU, a random accessmemory: RAM serving as a main storage device, a read-only memory: ROM,an HDD, or an SSD serving as an auxiliary storage device as illustratedas an example in FIG. 9 that will be described later. In addition,functions of the apparatus body 101 include an importance calculationportion 104, an inspection region determination portion 105 serving asan inspection region generation portion, an inspection processingportion 106 serving as an image inspection portion, and a setting tool107.

Among the functions described above, the importance calculation portion104 and the inspection region determination portion 105 are functionsfor determining an inspection region for image inspection performed bythe inspection processing portion 106. The setting tool 107 is afunction of supporting a user performing a setting operation of settinginformation required for inspection processing. An example of anembodiment of the setting tool 107, particularly, an example of a screendisplayed on the display apparatus 108 will be described with referenceto FIGS. 6A to 6C that will be described later.

For example, the functions described above are realized by software ofthe CPU constituting the apparatus body 101, or further by some kind ofhardware constituting the processing apparatus 100. For example, thesefunctions are realized by loading a computer program stored in theauxiliary storage device or the storage device 109 into the main storagedevice and executing the loaded computer program by the CPU.

To be noted, FIG. 1 merely illustrates an example of an apparatusconfiguration, and all or part of the image sensor 102, the displayapparatus 108, the storage device 109, and the input device 110 may beintegrated into the apparatus body 101. The apparatus body 101 may beconstituted by a personal computer: PC or a computer such as aslate-shaped terminal, or constituted by a dedicated chip or an on-boardcomputer.

Here, a more specific configuration of the control system of FIG. 1 isillustrated in FIG. 9. The control system of FIG. 9 can be constitutedby PC hardware, or the like, including a CPU 1601 serving as a maincontrol unit, a ROM 1602 serving as a storage device, and a RAM 1603. Animage processing program for the CPU 1601, constant information, and thelike, for realizing a control procedure that will be described later canbe stored in the ROM 1602. In addition, the RAM 1603 is used as a workarea of the CPU 1601 when executing the control procedure. Further, anexternal storage device 1606 is connected to the control system of FIG.9. The external storage device 1606 is constituted by an HDD, an SSD, anexternal storage device of another system that is mounted via a network,or the like.

An image processing program of the CPU 1601 for realizing the controlprocedure of the present exemplary embodiment illustrated in, forexample, FIG. 8, can be stored in a storage unit such as the externalstorage device 1606 constituted by an HDD, an SSD, or the like, the ROM1602, for example, an electrically erasable programmable read-onlymemory: EEPROM region, or the like. In this case, the image processingprogram of the CPU 1601 for realizing the control procedure that will bedescribed later can be supplied to the storage units described above viaa network interface 1607 and updated to a new program, that is, toanother program. Alternatively, the image processing program of the CPU1601 for realizing the control procedure that will be described latercan be supplied to the storage units described above via various storagemedia such as magnetic disks, optical disks, and flash memories, anddrive units therefor, and thus the content thereof can be updated. Thestorage units, storage portions, or storage devices of various kinds ina state of storing the image processing of the CPU 1601 for realizingthe control procedure constitute non-transitory computer-readablerecording media storing a control procedure of the present invention.

The image sensor 102 of FIG. 1, which is a camera, is connected to theCPU 1601. Although the image sensor 102 is illustrated as being directlyconnected to the CPU 1601 in FIG. 9 for the sake of simplicity, theimage sensor 102 may be connected to the CPU 1601 via a known camerainterface, or the like. In addition, the image sensor 102 may beconnected via a network interface 1607 and a network 1608.

The network interface 1607 can be configured by using, for example, acommunication standard of wired communication such as IEEE 802.3 or ofwireless communication such as IEEE 802.11 or 802.15. The CPU 1601 cancommunicate with other apparatuses 1104 and 1121 via the networkinterface 1607. The apparatuses 1104 and 1121 correspond to, forexample, integral control apparatuses such as a programmable logiccontroller: PLC, a sequencer, or the like, a management server, or thelike, disposed in a manufacturing line for production control andmanagement.

An operation portion 1604 and a display apparatus 1605 are arranged inthe control apparatus of FIG. 9 as a user interface: UI apparatus. Theoperation portion 1604 can be constituted by a terminal such as a handyterminal, a device such as a keyboard, a jog dial, or a pointing device,or a control terminal including these. For the display apparatus 1605,for example, a display apparatus of an arbitrary type may be usedinstead of the liquid crystal type as long as a data display screen thatwill be described later or a display screen similar thereto can bedisplayed thereon.

Inspection Processing

First, inspection processing performed by the processing apparatus 100will be described with reference to FIGS. 2 to 3E. FIG. 2 illustrates aflow of inspection processing of inspecting a target object on the basisof image processing, and FIGS. 3A to 3E illustrate how the target objectis inspected. To be noted, the flow of inspection processing will bedescribed below by taking inspection of presence/absence of, orinspection of the shape of an application region of, a coating materialapplied on a workpiece as an example.

In step S20 of FIG. 2, the image sensor 102 captures an image of aworkpiece 301 serving as a target object 103, and the captured imagedata is loaded into the apparatus body 101. To be noted, in the casewhen an image that has been captured in advance is present in theauxiliary storage device of the apparatus body 101 or the storage device109, the data to be inspected may be loaded from the auxiliary storagedevice or the storage device 109.

The image data of the target object loaded as described above isdisplayed on the display apparatus 108 as necessary. FIG. 3A illustratesan example of an original image without the coating material that isdisplayed on, for example, the display apparatus 108. In this originalimage, the workpiece 301 serving as a target object is captured. In thisexample, the target object is, for example, a circuit board, and animage capturing the workpiece 301, on which printed characters 302,wiring 303, an electronic chip 304, and the like, are disposed, isobtained as the original image. FIG. 3B illustrates an example of anoriginal image in which the circuit board on which the coating materialhas been already applied, and a coating material 305 is applied on theworkpiece 301.

In step S21, the inspection processing portion 106 loads requiredsetting information from the storage device 109. The setting informationat least includes an inspection region 306 and an inspection logic.

The inspection region 306 is a partial image region of the originalimage to be subjected to inspection processing using predetermined imageprocessing. In the example of FIG. 3C, the inspection region 306 isgenerated in a shape that approximately entirely covers the applicationregion of the coating material 305 but does not cover a regioncorresponding to the electronic chip 304. Such an inspection region 306can be set by, for example, manual setting in which a user designates aregion in a screen in which the captured image of the workpiece 301 isdisplayed. However, in the present exemplary embodiment, the size,position, shape, and the like, of the inspection region 306 can bedetermined by processing that will be described later. In addition, theinspection logic is information that defines the content of theinspection processing, and examples thereof include the kinds of featurevalue and determination method that are to be used for the inspection,and parameters and threshold values to be used for feature valueextraction and determination processing.

In step S22, the inspection processing portion 106 extracts a portioncorresponding to the inspection region from the original image by usingthe definition of the inspection region 306 loaded in step S21, and thusgenerates an inspection region image.

In this kind of inspection using an image, the inspection region 306 isdesirably set in such a size, position, and shape that only pixels to beinspected can be extracted without excess or deficiency. When excesspixels are included in the inspection region 306, the excess pixels maycause noises to degrade the inspection accuracy, and conversely, whenthe inspection region 306 is smaller than a range that should beinspected, the inspection may be incomplete.

However, in images in captured data, the position and shape of thetarget object are not necessarily always the same. For example, FIGS. 3Dand 3E illustrate an example of a target object in a differentworkpiece. In this example, printed characters 307 and an electronicchip 309 disposed on the workpiece are displaced and inclined, and theamount and shape of a coating material 308 are different. Therefore, asillustrated in FIG. 3E, an actual application region 310 of the coatingmaterial is displaced from an inspection region of the same shape as theinspection region 306 of FIG. 3C. In the case when there is apossibility that such change occurs, for example, for each capturedimage, there is a possibility that sufficient inspection cannot beperformed by only performing image processing on the inspection region.Therefore, in the present exemplary embodiment, an appropriateinspection region is set by using a plurality of images with or withoutthe coating material that are captured in advance as will be describedlater.

In step S23, the inspection processing portion 106 extracts a requiredfeature value in accordance with the inspection logic by using aninspection region image cut out from the inspection region 306 in stepS22. For example, as a feature value for inspecting presence/absence ofthe coating material 305, the area or a color feature value of theportion corresponding to the coating material 305 is extracted.

Next, in step S24, the inspection processing portion 106 performsinspection processing in accordance with the inspection logic. Forexample, in the case when the feature value of the area or color featurevalue of the coating material 305 obtained in step S23 is abnormal, itis determined that the coating material is absent.

In step S25, the inspection processing portion 106 performs outputprocessing of displaying an inspection result on the display apparatus108 or storing the inspection result in the storage device 109. In thismanner, the inspection processing on the workpiece 301 is completed.

The image processing apparatus configured as described above can be usedas an inspection apparatus that performs inspection processing of, forexample, inspecting presence/absence of a different member attached tothe workpiece or inspecting the application state of the coatingmaterial on the workpiece. Manufacturing apparatuses that performmanufacturing processes such as machining, assembly, and conveyance ofthe workpiece are placed in the manufacturing system, that is, themanufacturing line, by using the inspection result obtained by the imageprocessing apparatus, and the workpiece after completing a specific stepin the manufacturing processes is inspected by the image processingapparatus.

In such a manufacturing system, that is, manufacturing line, the imageprocessing apparatus of the present exemplary embodiment can be causedto repeatedly perform the inspection processing described above onworkpieces serving as target objects that are conveyed successively. Theinspection result of this can be, for example, input to themanufacturing apparatuses, and used for causing the manufacturingapparatuses to perform machining again or perform correction machiningon the basis of the inspection result. In addition, in the case when themanufacturing apparatus is a conveyance apparatus that switches aconveyance destination between good and defective products, theconveyance apparatus can be caused to perform the switching of theconveyance destination between good and defective products by using theinspection result obtained by the image processing apparatus describedabove.

Setting of Inspection Region

An operation of the processing apparatus 100 concerning setting of theinspection region will be described with reference to FIGS. 4 to 5F.FIG. 4 illustrates a flow of inspection region setting in the processingapparatus. In addition, FIGS. 5A to 5F are diagrams for describing amethod of generating an inspection region from image data.

In step S40 of FIG. 4, the inspection region determination portion 105obtains a group of images constituted by a plurality of images fordetermining the inspection region from the storage device 109.Alternatively, image data obtained by imaging the workpiece 301 by theimage sensor 102 may be used.

Here, for example, a workpiece such as a circuit board is set as atarget object for the inspection, and the application state of thecoating material on the target object is inspected by image processing.In this image inspection, OK determination indicating a good product,that is, inspection passing determination is made as a first inspectionresult, for example, in the case when the coating material is applied ina predetermined application region. This case corresponds to presence ofthe coating material. In addition, NG determination indicating adefective product, that is, inspection non-passing determination is madeas a second inspection result in the case when a predeterminedapplication state is not obtained. This case corresponds to absence ofthe coating material. To be noted, in the description below, a firstimage capturing a target object of OK determination indicating a goodproduct may be also referred to as an OK image, and a second imagecapturing a target object of NG determination indicating a defectiveproduct may be also referred to as an NG image.

In the present exemplary embodiment, the group of images read from thestorage device 109 in step S40 includes a plurality of OK imagescapturing a target object with the coating material, that is, a goodworkpiece, and a plurality of NG images capturing a target objectwithout the coating material, that is, a defective workpiece. Althoughthe number of one of the OK images and the NG images may be at leastone, it is preferable that the numbers of both of the OK images and theNG images that are input are two or more.

In FIG. 5A, 501 denotes an example of an OK image capturing a targetobject with the coating material, that is, a good workpiece. To benoted, the OK images and the NG images may be obtained by imaging targetobjects with the coating material, that is, good workpieces, and targetobjects without the coating material, that is, defective workpieces, inadvance. Inspection of whether the target objects captured in theseimages are good or defective products is performed by, for example,visual observation. Alternatively, a system in which, when inputtingimages capturing the target objects with/without the coating material,the images are displayed one by one on the display apparatus 108 or1605, and a user makes the OK/NG determination by using the input device110 may be employed. The OK images and the NG images are associated withflags, or the like, indicating the inspection results of OK/NG, and areloaded onto the storage device 109, for example, the RAM 1603, in aformat identifiable by the CPU 1601 in subsequent processing.

In step S41, the inspection region determination portion 105 loadssetting of an initial inspection region 502 illustrated in FIG. 5B. Theinitial inspection region 502 may be set by a user roughly designatingan inspection target portion, or setting the entire region of the imageas the initial inspection region. In FIG. 5C, 503 denotes an enlargedportion of the initial inspection region 502, and each block thereincorresponds to one pixel of the image.

In step S42, the inspection region determination portion 105 obtains aluminance value as a feature value for generating the inspection regionfrom each pixel address of the initial inspection region 502 of theimage, and loads the obtained luminance value onto the storage device109, for example, the RAM 1603. This processing is repeatedly performedon all of the input OK and NG images. The OK images correspond topresence of the coating material and the NG images correspond to absenceof the coating material.

In the case when the processing of obtaining the luminance valuedescribed above has been performed on all of the OK and NG images, thatis, in the case when the result of step S43 is Yes, the process proceedsto step S44, and the importance calculation portion 104 calculates theimportance of each pixel address on the basis of the luminance values ofthe OK and NG images. This identifiability value obtaining stepcorresponds to a function of the importance calculation portion 104serving as an identifiability value obtaining portion.

In the present exemplary embodiment, a specific “pixel address”corresponds to a position of the same specific field angle in an imageplane of the OK images corresponding to presence of the coating materialand the NG images corresponding to absence of the coating material. Oneblock in FIG. 5C or FIGS. 5E and 5F corresponds to the “pixel address”.

FIG. 5D illustrates how luminance values of a specific pixel in aplurality of OK images 501 and NG images 5011 are obtained as histogramsillustrated in a lower portion of FIG. 5D. Two histograms shown in lowerleft and lower right portions of FIG. 5D respectively correspond to aluminance distribution 506 of a specific pixel in the plurality of NGimages corresponding to absence of the coating material and a luminancedistribution 507 of the same specific pixel in the plurality of OKimages corresponding to presence of the coating material. This pixel isincluded in, for example, the predetermined application region of thecoating material, and an object of a different luminance is captured inthis pixel address between the case when the coating material is presentand the case when the coating material is absent. In addition, in thisexample, the coating material is a material reflection light on whichhas a lower luminance than that of a surface without the coatingmaterial such as the surface of the board, and the luminance values ofmedians of the two luminance histograms of the specific pixel are verydistant from each other. In addition, due to unevenness of the coatingmaterial, or the like, the variety of the luminance value of thespecific pixel in the OK images corresponding to presence of the coatingmaterial is greater than in the case of the NG images corresponding toabsence of the coating material in which approximately the same boardsurface, or the like, is captured. Conversely, in a specific pixeladdress where a board surface not related to the application region iscaptured, the distribution of the luminance histogram has a similarmedian value and a similar variance in both of the case of the OK imagescorresponding to presence of the coating material and the case of the NGimages corresponding to absence of the coating material.

In the present exemplary embodiment, the nature described above is used,and a high importance value serving as an identifiability value isassigned to a pixel whose feature value distribution, for example,luminance value distribution differs greatly between the OK imagescorresponding to presence of the coating material and the NG imagescorresponding to absence of the coating material.

In FIG. 5E, 505 denotes the importance value serving as anidentifiability value of each pixel address obtained on the basis ofluminance value data. The importance illustrated in FIG. 5E andcalculated for each pixel address of the entirety of the inspectionregion in this manner is held on the RAM.

The importance value calculated for each pixel address in this mannercorresponds to an identifiability value for identifying an OK imagecorresponding to presence of the coating material from an NG imagecorresponding to absence of the coating material. In the presentexemplary embodiment, a higher importance value serving as anidentifiability value is imparted to a pixel address when an OK imagecorresponding to presence of the coating material can be distinguishedbetter from an NG image corresponding to absence of the coatingmaterial.

Various methods for obtaining the importance Im can be considered. Forexample, as expressed in the following formula (1), a method of settinga luminance difference between an average luminance Lem of the OK imagesand an average luminance Lnm of the NG images as the importance can beconsidered.

Im=Lem−Lnm  (1)

In addition, a method of setting a value obtained by dividing theluminance difference between the average luminance Lem of thecoating-present images and the average luminance Lnm of thecoating-absent images by the sum of respective variance values Gem andσnm thereof as the importance as expressed in the following formula (2)can be considered.

Im=|Lem−Lnm|/(σem+σnm)  (2)

Also, a method of setting a value obtained by dividing the maximum valuemax(Lei−Lnj) of the luminance difference between the coating-presentimage and the coating-absent image by the minimum value min(Lei−Lnj)thereof as the importance as expressed in the following formula (3) canbe considered.

Im=max(Lei−Lnj)/min(Lei−Lnj)  (3)

Further, a method of performing determination analysis processing by adetermination analysis method of classifying the OK images and the NGimages, for example, Otsu's binarization, is performed, and setting themaximum value of a value obtained by dividing an in-class variance by aninter-class variance as expressed by the following formula (4) andserving as an evaluation standard as the importance can be considered.

Im=max(σ_(b) ²/σ_(w) ²)  (4)

To be noted, although a case when the luminance is used as a featurevalue in the importance calculation expressed by the formulae (1) to (4)described above has been described as an example, the importance can becalculated in a similar manner also in the case of using another featurevalue, for example, a hue value of the pixel. In addition, the methodsof importance calculation described above are merely examples, and anarbitrary calculation method can be used as long as the importance canbe calculated by using the luminance distributions in the coating-absentimage group and the coating-present image group as an indicator of theidentifiability value indicating how well the first and secondinspection results can be distinguished.

In step S45, the inspection region determination portion 105 serving asan inspection region generation portion generates a partial image regionincluding pixels of importance equal to or higher than a certain valueas an inspection region 508. FIG. 5F illustrates a case when a partialimage region including pixels whose importance calculated in FIG. 5E areforty or higher is generated as the inspection region 508. In step S46,the inspection region determination portion 105 stores the determinedinspection region 508 in the storage device 109.

As described above, in the processing steps executed by the inspectionregion determination portion 105, the luminance values in the initialinspection region are obtained by using the plurality of OK imagescorresponding to presence of the coating material and the plurality ofNG images corresponding to absence of the coating material. Then, by acalculation method such as ones expressed by the formulae (1) to (4)described above, the importance value serving as an identifiabilityvalue corresponding to the distribution state of the luminance value iscalculated for each pixel address. Then, a partial image regionincluding pixel addresses having importance values serving asidentifiability values satisfying a specific condition is determined asthe inspection region. For example, a partial image region includingpixel addresses having importance values serving as identifiabilityvalues equal to or higher than a certain value is determined as theinspection region.

Here, an example of a user interface constituting the setting tool 107for supporting a user in the case of performing the inspection regiongeneration described above will be described with reference to FIGS. 6Ato 6C. FIGS. 6A to 6C illustrate an example of configuration of adisplay screen of the display apparatus 108 or 1605 that can be operatedby a keyboard or a pointing device such as a mouse of the input device110 serving as the operation portion 1604.

When the setting tool 107 is activated, a setting screen illustrated inFIG. 6A is displayed on the display apparatus 108. This setting screenincludes an image window 600, an image registration button 601, and aninitial region loading button 602. In addition, this setting screenincludes, as user interface related to the importance, an importancecalculation method selection box 603, an importance threshold settingbox 604, and an importance threshold setting slide bar 605. Further,this setting screen includes, as user interface related to progress ofthe processing, a processing start button 606, a region determinationbutton 607, an image switching button 608, and a region display checkbox 609. To be noted, a matching button 610 is used for activatingidentification and correction processing of a position and a phase of atarget object in the image by using pattern matching that will bedescribed later in a second exemplary embodiment.

A user can perform selection from these buttons, input to a text/numberinput box, operation of the slide bar, and the like, by using a keyboardor a pointing device such as a mouse of the input device 110. To benoted, the setting screen of FIGS. 6A to 6C is merely an example, andany UI may be used as long as input of parameters and recognition of theinspection region that will be described below can be performed throughthe UI.

When the image registration button 601 is pressed, the setting tool 107inputs the group of images constituted by the plurality of OK imagescorresponding to presence of the coating material and the plurality ofNG images corresponding to absence of the coating material from thestorage device 109. One of the group of images input herein is displayedin the image window 600 of the setting screen as illustrated in FIG. 6A.

In addition, the user can switch the image displayed in the image window600 among the input group of images by pressing the image switchingbutton 608. In addition, when the initial region loading button 602 ispressed, the setting tool 107 loads data of the initial inspectionregion 502 preset for the image from the storage device 109. The formatof the data of the initial inspection region 502 may be any format, andthe entire region of the image, that is, the entire field angle may beused as the initial inspection region 502 as described above.

As illustrated in FIG. 6A, the inspection region 502 is superimposed onthe image in the image window 600 of the setting screen. In addition,ON/OFF of the superimposition of the initial inspection region 502 isswitched by operating the region display check box 609.

An importance calculation method is selected from the importancecalculation method selection box 603. The importance calculation methodselection box 603 can be implemented in the form of a pull-down orpull-up menu or a text input box. The user selects a differentimportance calculation method among, for example, the formulae (1) to(4) described above, by using the importance calculation methodselection box 603.

When the processing start button 606 is pressed, importance calculationprocessing on pixels in the initial inspection region 502 is performedfor the registered group of images. To be noted, an operation ofpressing the processing start button 606 without determining the initialinspection region 502 may be allowed. In this case, the entire region ofthe image is automatically set as the initial inspection region, and theimportance calculation processing is performed on each pixel in theentire region of the image. If the user interface is configured suchthat the initial inspection region 502 is selectable, a region in thefield of view that a user can determine as a region obviously notrequiring inspection can be excluded from the importance calculationprocessing, which leads to a possibility of accelerating the importancecalculation processing.

When the importance calculation processing is completed, as illustratedin FIG. 6B, a group of pixels having importance values serving asidentifiability values equal to or higher than a value designated viathe importance threshold setting box 604 is superimposed as aninspection region 620 on the image in the image window 600 of thesetting screen. In this case, an importance threshold value isconfigured to be changeable by a user rewriting a value in theimportance threshold setting box 604 or moving the importance thresholdsetting slide bar 605 to the left or the right. In addition, theimportance calculation method selection box 603 is configured such thatthe user can change the importance calculation method as describedabove. When the importance calculation method or the importancethreshold value is changed by such a GUI, as illustrated in FIG. 6C, theshape, position, and size of the inspection region 620 displayed in theimage window 600 are updated.

According to the user interface described above, the shape, position,and size of the generated inspection region 620 are automaticallyupdated in the displayed screen. With such a screen of the setting tool107, the user can recognize the shape, position, size, and the like, ofthe inspection region 620 at a glance.

To be noted, a user interface through which the inspection region 620can be deformed or another inspection region 620 can be added in thecase when a region the user desires to include in the inspection region620 is not included in the inspection region 620, even if the importancecalculation method or the importance threshold value is changed, may befurther provided. Such an inspection region editing user interface canbe implemented by a GUI for deforming or adding the inspection region620 through an operation of a pointing device such as a mouse.

When a desired inspection region 620 is obtained in the image window 600in this manner, the user confirms the inspection region 620 by theregion determination button 607. When this operation is performed,inspection region definition information corresponding to the inspectionregion 620 that is currently superimposed on the image in the imagewindow 600 is generated. The data format of this inspection regiondefinition information can be arbitrarily selected, and the generatedinspection region definition information is stored in the storage device109, the external storage device 1606, or the like.

As described above, according to the present exemplary embodiment, theimportance of each pixel of the image captured by the image sensor 102for the inspection can be calculated by using OK images and NG images ofa certain number prepared in advance. This importance value correspondsto an identifiability value for identifying an OK image or an NG imagerespectively corresponding to the first and second inspection results inthe position, that is, address, of each pixel. Further, an inspectionregion including pixels whose importance satisfies a predeterminedcondition, for example, pixels whose importance values serving asidentifiability values are equal to or greater than a predeterminedthreshold value can be automatically generated. Therefore, even in thecase of inspecting a target object of a different shape each time, aportion of the background whose color is similar to that of the targetobject is not automatically set as the inspection region, and onlypixels that differ greatly between OK images and NG images can be set asthe inspection region, by preparing a sufficient number of images.Therefore, removal of the background portion, or the like, in theinspection region does not have to be performed in subsequent inspectionprocessing, and, thus, quick, highly precise, and robust imageinspection can be performed. That is, according to the present exemplaryembodiment, an appropriate inspection region with which erroneousdetection is unlikely to occur can be set quickly and automatically evenin the case of inspecting a target object whose position, or the like,in the captured image is different each time.

According to the image processing method and the image processingapparatus of the exemplary embodiment, quick, highly precise, and robustinspection can be executed by using the appropriately set inspectionregion. A highly reliable manufacturing line can be established byarranging manufacturing apparatuses that control machining, assembly,conveyance, and the like, of the workpiece, by using results ofinspection performed by the image processing method and the imageprocessing apparatus.

Second Exemplary Embodiment

A second exemplary embodiment of the present invention will be describedwith reference to FIGS. 7 to 8B. The difference from the first exemplaryembodiment is that correction in the case when the position of theworkpiece including the target object, that is, the position and phaseof the target object is different for each captured image is included.In the second exemplary embodiment, processing of specifying andcorrecting the position and phase of the target object is added. Theother elements and processing are the same as in the first exemplaryembodiment. In the present exemplary embodiment, for example, an imageprocessing technique such as pattern matching processing or affinetransformation is used for the specification and correction of theposition and phase.

FIG. 7 illustrates processing of the second exemplary embodiment thatshould replace steps S42 and S43 of the flow of the first exemplaryembodiment illustrated in FIG. 4. As has been described for steps S40and S41 of FIG. 4, the inspection region determination portion 105 ofFIG. 1 loads the target object image and the initial inspection regionsimilarly to the first exemplary embodiment.

Next, in step S70 of FIG. 7, the inspection region determination portion105 loads matching parameters for matching processing. Examples of thematching parameters include a score threshold value, a searching angle,a searching scale, and a searching region of pattern matching.

Next, in step S71, the inspection region determination portion 105performs pattern matching processing. This pattern matching processingis matching processing between the image subjected to processing, whichis one of the OK and NG images, and a shape pattern of a template imageor model image of the target object or a part of the target object thatis prepared in advance. The details of this processing will be describedlater.

Next, in step S72, the inspection region determination portion 105performs affine transformation on the image on the basis of the resultof the matching. As a result of this, the positions and phases of thetarget object in the OK images or NG images are aligned on the basis ofthe template image.

In step S73, the inspection region determination portion 105 obtainsluminance values of the initial inspection region 502 of the image andloads the luminance values onto the RAM. This processing is repeatedlyperformed on all images. When the processing is completed for all theimages, that is, when the result of step S74 is Yes, the processproceeds to step S44 of FIG. 4. The subsequent steps are the same as inthe first exemplary embodiment.

In the manner described above, the positions and phases of the targetobject in the OK images or NG images can be specified and, for example,corrected to coincide with one another on the basis of the templateimage. By this correction processing, it is guaranteed that a pixel ofthe same pixel address in the images captures the same portion of thetarget object. Therefore, the importance value serving as anidentifiability value of each pixel can be calculated more precisely.

Pattern Matching

The pattern matching processing performed in step S71 of FIG. 7 can beexecuted via, for example, edge detection processing as illustrated inFIGS. 8A and 8B. In image data 801 illustrated in FIG. 8A, a targetobject 802 schematically illustrated as a triangle is captured. In thepattern matching using edge extraction, an edge intensity in a pixel ofinterest 803 is evaluated. FIG. 8B illustrates edge intensity that isedge information characterizing the nature of the edge at the pixel ofinterest 803 included in one of oblique sides of the target object 802illustrated in FIG. 8A.

The pattern matching processing using edge intensity information isprocessing of searching for a position at which similarity to thetemplate image serving as a standard image registered in advance is thehighest in the searching region. In the case when the similarity betweenthe input image and the standard image at the searched position, thatis, the matching score, is higher than a score threshold value, thematching processing is successful. Then, among the positions where thematching processing is successful, a position with the highest score isoutput as a matching position. Pattern matching processing between anedge image extracted from a workpiece image and the standard image willbe described herein.

As illustrated in FIG. 8A, among all the pixels of the workpiece image,pixels whose gradient intensity E is equal to or higher than a thresholdvalue, that is, an edge is extracted. In this edge extraction, thegradient intensity and gradient direction of luminance are calculatedfor all the pixels of the workpiece image. The gradient intensity iscalculated by using a sobel filter of an x direction and a sobel filterof a y direction. An x-axis direction gradient intensity Ex and a y-axisdirection gradient intensity Ey are calculated for each pixel.

Then, as illustrated in FIG. 8B, the final gradient intensity E of thetarget object in each image, which is an oblique side of the triangle,can be calculated as a square root of a sum of squares of the x-axisdirection gradient intensity Ex and the y-axis direction gradientintensity Ey, for example, as expressed by the following formula (5).

E=√{square root over (Ex ² +Ey ²)}  (5)

In addition, at this time, a gradient direction θ can be calculated bythe following formula (6) by using the x-axis direction gradientintensity Ex and the y-axis direction gradient intensity Ey.

$\begin{matrix}{\theta = {\tan^{- 1}\left( \frac{Ey}{Ex} \right)}} & (6)\end{matrix}$

Then, the edge of the workpiece image is extracted by selecting pixelswhose gradient intensity E is equal to or greater than a predeterminedthreshold value among all the pixels of the workpiece image. Next,pattern matching processing is performed between the extracted edge andone standard image to evaluate the standard image. Further, the matchingscore Sij of the standard image at a detected position (i, j) can becalculated pixel by pixel by using the following formula (7) for all thepixels of the extracted edge in the image data.

$\begin{matrix}{S_{ij} = {\frac{1}{N}{\sum\limits_{k = 0}^{N - 1}S_{k}}}} & (7)\end{matrix}$

Here, a local score Sk is a score calculated for each model edge pointof a candidate model, and is calculated by the following formula. In thefollowing formula (8), θTk is a gradient direction of each edge point ofthe extracted edge, and θMk is a gradient direction of each edge pointof the model edge.

S _(k)=CoS(θ_(Tk)−θ_(Mk))  (8)

Here, the local score Sk and the matching score Sij each take a value of−1 to +1, and the value being closer to one means that the level ofpattern matching is higher.

In this manner, pattern matching of S71 of FIG. 7 can be performed viaedge extraction. The affine transformation of step S72 of FIG. 7performed subsequently to the pattern matching is performed aftersetting parameters related to translational movement and rotation of theimage to such values that the matching score with the template imagebecomes the highest. In this manner, the position and phase of thetarget object can be specified and corrected such that the position andphase of the target object is the same in all the images.

As described above, the inspection region generation can be performedafter correcting the position and phase of the target object captured inthe OK images or NG images by using the pattern matching, and thus theinspection region that can be used more highly precisely and morerobustly can be generated. That is, according to the second exemplaryembodiment, an effect similar to that of the first exemplary embodimentcan be achieved with higher precision. The process illustrated in FIG. 7is a process to which processing of specifying and correcting theposition and phase of the workpiece is added, and can be preferablyapplied to a case when the position of the workpiece including thetarget object differs between images.

The processing of specifying and correcting the position and phase ofthe target object using the pattern matching of the second exemplaryembodiment may be configured to be activated and deactivated inaccordance with various conditions. For example, the matching button 610illustrated in FIGS. 6A to 6C is provided as an operation portion fordetermining whether or not the processing of specifying and correctingthe position and phase of the target object using pattern matching is tobe performed when generating the inspection region. By preparing a userinterface like this matching button 610, the user can determine whetheror not to perform the processing of specifying and correcting theposition and phase of the target object using pattern matching, inaccordance with the situation when the inspection region is generated.For example, the activation/deactivation operation of the matchingbutton 610 can be selected in consideration of change in the positionand phase of the workpiece in the manufacturing line where theinspection is performed, change in the position and phase of theworkpiece captured in the OK images or NG images used for generation ofthe inspection region, and the like.

To be noted, in the second exemplary embodiment described above, shapematching focused on the edge has been described as an example offeature-based matching processing. However, the method of specifying theposition and phase of the target object in the image is not limited tothis. For example, feature-based matching may be used for specifying andcorrecting the position and phase of the target object, or matchingprocessing focusing on a corner, or the like, may be used. In addition,area-based matching may be used for specifying and correcting theposition and phase of the target object, or a method of specifying theposition and phase of the workpiece by performing binarization whilechanging the threshold value for the binarization may be used.

The exemplary embodiments described above are specific examples of thepresent invention, and the scope of the present invention should not belimited to the specific examples.

Although the luminance value of the pixel has been used as a featurevalue used for calculation of the importance value serving as anidentifiability value of the pixel in the exemplary embodimentsdescribed above, a two-dimensional histogram of color, that is, a colorspace, or a multi-dimensional histogram of color including hue,saturation, and luminance may be used.

To be noted, although all the pixels whose importance is higher than athreshold value has been included in the inspection region in theexemplary embodiments described above, only a region of the largest areamay be set as the inspection region after joining and dividing regions,or the user may be let designate the inspection region. In addition,noise reduction processing may be performed. For example, openingprocessing or closing processing that appropriately performs a requiredprocess in accordance with the purpose may be added before or after theimage processing or the inspection region determination processing. Inaddition, although an original image has been used in the exemplaryembodiments described above, a method of reducing the amount ofcalculation by using a reduced image may be employed. In addition,although the importance value serving as an identifiability value hasbeen used for determining the inspection region in the first and secondexemplary embodiments described above, processing of, by using the sameindicator, treating a portion whose importance value serving as anidentifiability value is low as a standard region, for example, a regionwhere the luminance value does not change between the coating-presentimage and the coating-absent image may be performed.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or an apparatus that reads out and executescomputer executable instructions (e.g., one or more programs) recordedon a storage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., an application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., a central processingunit (CPU), or a micro processing unit (MPU)) and may include a networkof separate computers or separate processors to read out and to executethe computer executable instructions. The computer executableinstructions may be provided to the computer, for example, from anetwork or the storage medium. The storage medium may include, forexample, one or more of a hard disk, a random-access memory (RAM), aread only memory (ROM), a storage of distributed computing systems, anoptical disk (such as a compact disc (CD), a digital versatile disc(DVD), or a Blu-ray Disc (BD)™) a flash memory device, a memory card,and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

1. (canceled)
 2. An image processing method comprising: inputting afirst image to an image processing apparatus, the first image beingassociated with a first inspection result; inputting a second image tothe image processing apparatus, the second image being associated with asecond inspection result different from the first inspection result;inputting a third image to the image processing apparatus; and setting aportion of the third image as an area to be subjected to imageprocessing, by the image processing apparatus, based on the first imageand the second image.
 3. The image processing method according to claim2, further comprising configuring, by the image processing apparatus, aregion determination portion based on the first image and the secondimage, wherein, in the setting, the image processing apparatus uses theregion determination portion to set the portion of the third image. 4.The image processing method according to claim 2, further comprisingimage processing, by the image processing apparatus, the portion of thethird image.
 5. The image processing method according to claim 2,wherein the image processing apparatus is configured to change acondition of the setting on a basis of the first image and the secondimage.
 6. The image processing method according to claim 2, wherein theimage processing apparatus is configured to be customized by a conditionof the setting by a user.
 7. The image processing method according toclaim 2, wherein the image processing apparatus is configured to becustomized by the area to be subjected to image processing by a user. 8.The image processing method according to claim 2, wherein, in theconfiguring, the first image and the second image are displayed on adisplay apparatus.
 9. The image processing method according to claim 2,further comprising obtaining histograms, by the image processingapparatus, based on the first image and the second image.
 10. The imageprocessing method according to claim 2, wherein the image processingapparatus is configured to correct at least one of the first image, thesecond image, and the third image.
 11. The image processing methodaccording to claim 10, wherein the image processing apparatus is furtherconfigured to perform an image correction and a pattern matchingprocessing.
 12. The image processing method according to claim 2,wherein the image processing apparatus is configured to display asetting screen on a display apparatus.
 13. The image processing methodaccording to claim 12, wherein the setting screen includes at least oneof an image window, an image registration button, and an initial regionloading button.
 14. The image processing method according to claim 12,wherein the setting screen includes at least one of an importancecalculation method selection box, an importance threshold setting box,and an importance threshold setting slide bar.
 15. The image processingmethod according to claim 12, wherein the setting screen includes atleast one of a processing start button, a region determination button,an image switching button, and a region display check box.